Papers with language modeling tasks
Regularized Training of Nearest Neighbor Language Models (2022.naacl-srw)
Copied to clipboard
| Challenge: | kNN-LM uses pre-trained language models and an exhaustive knn search to improve performance. |
| Approach: | They build upon kNN-LM, which uses a pre-trained language model and a knn search through the training data to achieve state-of-the-art results. |
| Outcome: | The proposed method improves on language modeling tasks on WIKI-2 and WIKI-103. |
Improving and Simplifying Template-Based Named Entity Recognition (2023.eacl-srw)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is traditionally approached as a sequence labeling task where a tag is predicted for each token. |
| Approach: | They propose to convert a Named Entity Recognition task into a seq2seq task by generating synthetic sentences using templates. |
| Outcome: | The proposed model outperforms the current state-of-the-art approach in resource-rich, low resource and domain transfer settings and the negative examples play an important role in its performance. |
How much complexity does an RNN architecture need to learn syntax-sensitive dependencies? (2020.acl-srw)
Copied to clipboard
| Challenge: | Long-term memory (LSTM) networks are capable of encapsulating long-range dependencies . but simple recurrent networks (SRNs) have been less successful at capturing long-term dependencies and loci of grammatical errors in an unsupervised setting. |
| Approach: | They propose a new architecture that incorporates the decaying nature of neuronal activations and models the excitatory and inhibitory connections in a population of neurons. |
| Outcome: | The proposed architecture shows competitive performance relative to LSTMs on subject-verb agreement, sentence grammaticality, and language modeling tasks. |
Training-free Neural Architecture Search for RNNs and Transformers (2023.acl-long)
Copied to clipboard
| Challenge: | Neural architecture search (NAS) has allowed for the automatic creation of new and effective neural network architectures. |
| Approach: | They develop a new NAS metric that predicts the trained performance of an RNN architecture and significantly outperforms existing NAS metrics. |
| Outcome: | The proposed metric outperforms existing training-free metrics on the NAS-Bench-NLP benchmark. |
RankNAS: Efficient Neural Architecture Search by Pairwise Ranking (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods require training millions of architectures to estimate the accuracy of the search results. |
| Approach: | They propose a performance ranking method (RankNAS) that uses pairwise ranking and search space pruning to enlarge the search space. |
| Outcome: | The proposed method significantly accelerates NAS through pairwise ranking and search space pruning. |
FlowPrior: Learning Expressive Priors for Latent Variable Sentence Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing training strategies are not effective for learning rich priors, we propose adding the importance-sampled log marginal likelihood as a second term to the standard VAE objective. |
| Approach: | They propose to add importance-sampled log marginal likelihood to standard VAE objective to help when learning the prior. |
| Outcome: | The proposed model improves language modeling tasks compared to baselines. |
Anisotropy is Not Inherent to Transformers (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing work has shown that Transformer embeddings are anisotropic, which is called the representation degradation problem. |
| Approach: | They identify a set of Transformer models with isotropic embedding spaces, the large Pythia models. |
| Outcome: | The proposed model sets show that isotropic models do not develop as previously theorized. |
Meta-Transfer Learning for Code-Switched Speech Recognition (2020.acl-main)
Copied to clipboard
| Challenge: | Increasing number of people in the world today speak a mixed-language as a result of being multilingual. |
| Approach: | They propose a method to transfer learn on a code-switched speech recognition system by extracting information from high-resource monolingual datasets. |
| Outcome: | The proposed model outperforms baselines on speech recognition and language modeling tasks and is faster to converge. |
EmByte: Decomposition and Compression Learning for Small yet Private NLP (2025.findings-emnlp)
Copied to clipboard
| Challenge: | EMBYTE is a byte-level tokenization model that reduces embedding parameters by up to 94% . it is also resilient to privacy threats such as gradient inversion attacks . |
| Approach: | EMBYTE is a byte-level tokenization model that decomposes subwords into fine-grained byte embeddings and then compresses them via neural projection. |
| Outcome: | EMBYTE achieves substantial embedding compression while preserving accuracy and enhancing privacy. |
Automatic Label Sequence Generation for Prompting Sequence-to-sequence Models (2022.coling-1)
Copied to clipboard
| Challenge: | Prompting has shown to be sample efficient compared to fine-tuning with pre-trained models. |
| Approach: | They propose a fully automatic prompting method that uses natural language prompts on sequence-to-sequence models and a beam search method to generate a large amount of label sequence candidates. |
| Outcome: | The proposed method significantly outperforms other no-manual-design methods on single label words and generates large amount of label sequence candidates. |
The Devil in Linear Transformer (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing linear transformers suffer from performance degradations on various tasks and corpus. |
| Approach: | They propose a new linear attention that replaces scaling with a normalization to stabilize gradients and confine attention to neighbouring tokens in early layers. |
| Outcome: | The proposed model outperforms vanilla transformers on the long-range arena benchmark while being significantly more space-time efficient. |
Pyramidal Recurrent Unit for Language Modeling (D18-1)
Copied to clipboard
| Challenge: | Long short term memory units are powerful tools for language modeling, but their performance can be limited by the number of parameters. |
| Approach: | They propose a pyramidal recurrent unit which enables learning representations in high dimensional space with more generalization power and fewer parameters. |
| Outcome: | The proposed model outperforms existing models with different gating mechanisms and transformations on word-level language modeling tasks. |
Structured Pruning of Large Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in language modeling have led to remarkable improvements on a variety of tasks. |
| Approach: | They propose a generic, structured pruning approach by parameterizing each weight matrix and adaptively removing rank-1 components during training. |
| Outcome: | The proposed method outperforms unstructured pruning and block pruning on language modeling tasks while achieving speedups during training and inference. |
Transformers to Learn Hierarchical Contexts in Multiparty Dialogue for Span-based Question Answering (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to embedding in multiparty dialogues are poor for span-based question answering (QA) |
| Approach: | They propose a novel approach to transformers that learns hierarchical representations in multiparty dialogue. |
| Outcome: | The proposed model improves on the FriendsQA dataset by 3.8% and 1.4% over the two state-of-the-art models. |
Linear Transformers with Learnable Kernel Functions are Better In-Context Models (2024.acl-long)
Copied to clipboard
Yaroslav Aksenov, Nikita Balagansky, Sofia Lo Cicero Vaina, Boris Shaposhnikov, Alexey Gorbatovski, Daniil Gavrilov
| Challenge: | Current Language Models (LMs) lack essential In-Context Learning capabilities, a domain where the Transformer excels. |
| Approach: | They propose a Linear Transformer with a kernel inspired by the Taylor expansion of exponential functions, augmented by convolutional networks. |
| Outcome: | The proposed model amplifies its In-Context Learning abilities on the Pile dataset. |
Flashback: Memory Mechanism for Enhancing Memory Efficiency and Speed in Deep Sequential Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing deep sequential processing models have problems with memory degradation and inaccurate gradient backpropagation. |
| Approach: | They propose a Flashback property that preserves memory as an identity mapping until it is overwritten by a hidden state at a different time step. |
| Outcome: | The proposed model can be implemented in Transformers and Mamba, and it performs well. |
Interpolated Spectral NGram Language Models (P19-1)
Copied to clipboard
| Challenge: | Spectral methods for learning weighted non-deterministic automata have nice theoretical and algorithmic properties, but they have been challenging to get competitive results on language modeling tasks. |
| Approach: | They propose a method for scaling up spectral learning and use interpolated predictions that are optimized to maximize perplexity. |
| Outcome: | The proposed method matches state-of-the-art ngram models while being very fast to train. |
When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute (2021.emnlp-main)
Copied to clipboard
| Challenge: | Large language models are difficult to train because of the growing computation time and cost. |
| Approach: | They propose a highly-efficient architecture that combines fast recurrence and attention for sequence modeling. |
| Outcome: | The proposed model achieves state-of-the-art on a Wiki-103 and Billion Word datasets using 1.6 days of training on an 8-GPU machine. |
Unraveling Feature Extraction Mechanisms in Neural Networks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Neural networks have become indispensable across a variety of natural language processing tasks. |
| Approach: | They propose a theoretical approach based on Neural Tangent Kernels to investigate neural networks' internal mechanisms. |
| Outcome: | The proposed approach can be applied to analyze language modeling tasks . it shows that the choice of activation function can affect feature extraction . |
Accelerating Toeplitz Neural Network with Constant-time Inference Complexity (2023.emnlp-main)
Copied to clipboard
| Challenge: | Toeplitz Neural Networks outperform commonly used Transformer-based models while benefiting from log-linear space-time complexities. |
| Approach: | They propose to convert TNNs to SSMs during inference to combine strengths of TNN and SSM approaches. |
| Outcome: | The proposed method outperforms most Transformer-based models while retaining the advantage of constant inference complexity. |
How do Transformer Embeddings Represent Compositions? A Functional Analysis (2025.findings-acl)
Copied to clipboard
| Challenge: | Despite the popularity of transformer-based models, little is known about how they represent compound words and whether they are compositional. |
| Approach: | They evaluate compositionality in mistral, OpenAI Large, and Google embedding models and compare them with BERT. |
| Outcome: | The proposed models perform best in addition, multiplication, dilation, regression, and the classic vector addition model performs almost as well as any other model. |
Unsupervised Morphological Tree Tokenizer (2025.findings-acl)
Copied to clipboard
| Challenge: | Conventional statistical tokenizers often disrupt constituent boundaries within words, thereby corrupting semantic information. |
| Approach: | They propose a method that uses morphological structure guidance to induce character-level structures of words by training a deep model. |
| Outcome: | Empirical results show that the proposed method retains complete morphemes and outperforms existing methods on morphological segmentation and language modeling tasks. |
Projective Methods for Mitigating Gender Bias in Pre-trained Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Mitigating gender bias in NLP has a long history tied to debiasing static word embeddings. |
| Approach: | They propose a masked language modelling task where content is developed around known social stereotypes and a projective debiasing method is used to reduce bias. |
| Outcome: | The proposed methods reduce intrinsic bias and mitigat observed bias in a downstream setting, but the two outcomes are not necessarily correlated. |
Targeted Syntactic Evaluation on the Chomsky Hierarchy (2024.lrec-main)
Copied to clipboard
| Challenge: | a novel evaluation paradigm for targeted syntactic evaluations is proposed . we create formal languages that abstract four syntaktic phenomena in natural languages . |
| Approach: | They propose a new evaluation paradigm for Targeted Syntactic Evaluations . they create formal languages that abstract syntactical phenomena in natural languages . |
| Outcome: | The proposed evaluation paradigm evaluates language models on language modeling tasks . it shows that they can capture the structural patterns of the (Adj)n NP type formal language . |
Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work shows that byte-pair encoding (BPE) tokenization uses a learned merge list to iteratively combine subword units into tokens during inference time. |
| Approach: | They propose to use a standard byte-pair encoding algorithm to pair a learned token vocabulary with a detailed merge list to compress text. |
| Outcome: | The proposed algorithms differ from the encoding process during training and show that the targetted deviation from merge lists exhibits significant degradation in language model performance. |